Stream memory leak

Mathias Mamsch - Sat Feb 16 16:11:03 EST 2013

Normally I use an eval_ context for cleaning up the streams, if I need to create a lot of them. In most cases you can get along, by keeping the stream open and just use flush (Stream) to make sure, that the content is indeed written. Anyway, if you need to resort to reading and writing a lot of files, you must use the eval context, for example as follows:
 

Buffer gBufWriteTextContentBuffer = create()
Buffer gBufWriteTextFilenameBuffer = create()
 
string gStrWriteBufferCode = "
    Buffer bufWriteTextContentBuffer  = (addr_ " ((addr_ gBufWriteTextContentBuffer) int) ") Buffer
    Buffer bufWriteTextFilenameBuffer = (addr_ " ((addr_ gBufWriteTextFilenameBuffer) int) ") Buffer
 
    // Create file and write contents; and catch errors if any
    noError
    
    // Create file for write access
    Stream os = write( bufWriteTextFilenameBuffer \"\")
    
    // Write needed contents to the batch file stream:
    os << bufWriteTextContentBuffer
    
    // Save stream contents to file and close it
    flush( os )
    close( os )
    
    return_ lastError
"
 
string writeWithoutLeak (string file_Name, Buffer fileContent) {
    gBufWriteTextFilenameBuffer = file_Name
    gBufWriteTextContentBuffer  = tempStringOf fileContent
    noError()    
    string msgEval = eval_ gStrWriteBufferCode
    string strErr = lastError()
    if (msgEval != "") return msgEval 
    return strErr 
}

Note that the code string is a constant here, due to the use of Buffers to marshal both fileName and content into the eval_ context. Note that this way you need to copy the fileContent buffer, so if you write a lot of large files, you will be quicker by using a buffer reference / pointer instead.

You can do the same for reading a file, which is left as an excercise ;-) Note that for marshalling an integer from an to an eval_ context you do:

int gFunctionResult = 0 
 
string strCode = " 
      int &retVal = addr_ " ((addr_ (&gReadFileRetVal)) int) "
      // ... your code ... 
      retVal = 4 // set your return value
"

again the string code stays static, so you do not need to litter the string table instead of leaking objects. Hope that helps, regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

SystemAdmin - Mon Feb 18 02:08:28 EST 2013
Hi Mathias

What is "a lot of files"?

1.000, 10.000 or 100.000?

Best regards
Wolfgang

Mathias Mamsch - Mon Feb 18 05:30:03 EST 2013
Well, every time you open a Stream you will add an object to the list of allocated objects, which you cannot get rid of anymore. These objects will degrade the performance of your program more and more, from my experience starting with about 1000 objects.

I remember something, that the time for allocation increased linearly from 10000 objects to 50000 objects by factor 1000, but the real slow down of the code depends on the relative speed of code that contains the allocation.

There are some posts on the forum, that contain discussions about the slowdown and code to demonstrate the slowdown:

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14672594&#14672594

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14775320&#14775320

If 10000 objects are much or few depends on the code. A script that allocates a buffer for each object will slow down for large modules. A script that will append to a log file for each object of a module too.

Regards, Mathias

SystemAdmin - Mon Feb 18 19:32:33 EST 2013
Mathias,
Your code is most appreciated!!! It works great with one tweak. I changed "Stream os = write(...)" to Stream os = append(..) so the text file was not overwritten each time the writeWithoutLeak procedure was called.
I admit I need a better understanding of eval_ and addr_ any suggestions where I can get more information?

Thanks for your help!

llandale - Tue Feb 19 09:32:28 EST 2013
Wikipedia "VooDoo".

SystemAdmin - Mon Feb 18 19:32:33 EST 2013
Mathias,
Your code is most appreciated!!! It works great with one tweak. I changed "Stream os = write(...)" to Stream os = append(..) so the text file was not overwritten each time the writeWithoutLeak procedure was called.
I admit I need a better understanding of eval_ and addr_ any suggestions where I can get more information?

Thanks for your help!

Mathias Mamsch - Tue Feb 19 11:19:37 EST 2013
I will make a new post for that. Regards, Mathias